Associating attribute seeds of regression test cases with breakpoint value-based fingerprints

Systems, methods, and computer-readable media are described for associating regression test cases with corresponding test fingerprints via a translation table or other suitable storage mechanism. A test fingerprint indicates a collection of breakpoints encountered as part of traversal of a code path during execution of a test case, and thus, provides an indication of an extent of code coverage of the test case. Test fingerprints can be evaluated to determine cumulative code coverage of sets of test cases. Specific sets of test cases can be selected based on the cumulative code coverage they provide. Sets of test cases that provide greater cumulative code coverage and/or that target specific areas of interest can be prioritized for execution particularly when having to adhere to execution constraints.

RELATED APPLICATIONS

This application which claims the benefits of multiple legally related applications including Application titled “FAULT DETECTION AND LOCALIZATION TO GENERATE FAILING TEST CASES USING COMBINATORIAL TEST DESIGN TECHNIQUES” assigned Ser. No. 16/256,388, filed Jan. 24, 2019; Application titled “FAULT DETECTION AND LOCALIZATION USING COMBINATORIAL TEST DESIGN TECHNIQUES WHILE ADHERING TO ARCHITECTURAL RESTRICTIONS” assigned Ser. No. 16/256,391, filed Jan. 24, 2019; Application titled “COMBINATORIC SET COMPLETION THROUGH UNIQUE TEST CASE GENERATION” assigned Ser. No. 16/256,394, filed Jan. 24, 2019; Application titled “TEST SPACE ANALYSIS ACROSS MULTIPLE COMBINATORIC MODELS” assigned Ser. No. 16/256,179, filed Jan. 24, 2019; Application titled “CHAMPION TESTCASE GENERATION” assigned Ser. No. 16/256,949, filed Jan. 24, 2019; Application titled “REGRESSION TEST FINGERPRINTS BASED ON BREAKPOINT VALUES,” assigned Ser. No. 16/440,404, filed Jun. 13, 2019; Application titled “SELF HEALING SOFTWARE UTILIZING REGRESSION TEST FINGERPRINTS,” not yet filed; Application titled “REDUCTION OF TEST INFRASTRUCTURE,” not yet filed; Application titled “BREAKPOINT VALUE-BASED VERSION CONTROL,” not yet filed; Application titled “FAULT DETECTION USING BREAKPOINT VALUE-BASED FINGERPRINTS OF FAILING REGRESSION TEST CASES,” not yet filed; and Application titled “CUSTOMIZABLE TEST SET SELECTION USING CODE FLOW TREES,” not yet filed, with all the benefits accruing therefrom, the contents of which in its entirety are herein incorporated by reference.

BACKGROUND

The present invention relates generally to regression testing, and more particularly, to generating, populating, and utilizing a translation table or other suitable storage mechanism that associates attribute seeds of regression tests with breakpoint value-based fingerprints of the regression tests.

Breakpoints are intentional stopping or pausing points in a program, typically inserted for the purposes of debugging. More generally, a breakpoint is a means of acquiring knowledge about a program's execution. When a breakpoint is reached during execution, a test environment (e.g., general purpose registers, memory, logs, files, etc.) to attempt to determine whether the program is functioning as expected. Breakpoints are commonly used to interrupt a running program immediately before execution of particular program instructions, in which case, they may be referred to as instruction breakpoints.

Combinatorial Test Design (CTD) is a testing methodology that seeks to increase test space coverage for a System Under Test (SUT) through the use of automated algorithms. These algorithms identify input patterns that are most likely to locate problems in the SUT, thereby reducing the amount of time required for a tester to build test cases and an automation framework. CTD is well-adapted for projects that require numerous variations on static input vectors to properly test various system states and logic pathways, which would otherwise be extremely cumbersome for a human tester.

CTD provides enhanced testing efficiency over manual testing through the use of automated algorithms. CTD methodologies, however, still suffer from a number of technical drawbacks, technical solutions to which are described herein.

SUMMARY

In one or more example embodiments, a method for associating a particular test case with a particular test fingerprint indicative of code coverage of the particular test case is disclosed. The method includes executing the particular test case. During execution of the particular test, a code path traversed during execution is determined and a collection of breakpoints encountered during traversal of the code path is determined. The method further includes determining the particular test fingerprint corresponding to the particular test case based at least in part on the collection of breakpoints, determining an attribute seed associated with the particular test case, and storing an association between the attribute seed and the particular test fingerprint.

In one or more other example embodiments, a system for associating a particular test case with a particular test fingerprint indicative of code coverage of the particular test case is disclosed. The system includes at least one memory storing computer-executable instructions and at least one processor configured to access the at least one memory and execute the computer-executable instructions to perform a set of operations. The operations include executing the particular test case. During execution of the particular test, a code path traversed during execution is determined and a collection of breakpoints encountered during traversal of the code path is determined. The operations further include determining the particular test fingerprint corresponding to the particular test case based at least in part on the collection of breakpoints, determining an attribute seed associated with the particular test case, and storing an association between the attribute seed and the particular test fingerprint.

In one or more other example embodiments, a computer program product for associating a particular test case with a particular test fingerprint indicative of code coverage of the particular test case is disclosed. The computer program product includes a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a processing circuit to cause the processing circuit to perform a method. The method includes executing the particular test case. During execution of the particular test, a code path traversed during execution is determined and a collection of breakpoints encountered during traversal of the code path is determined. The method further includes determining the particular test fingerprint corresponding to the particular test case based at least in part on the collection of breakpoints, determining an attribute seed associated with the particular test case, and storing an association between the attribute seed and the particular test fingerprint.

DETAILED DESCRIPTION

Example embodiments of the invention relate to, among other things, devices, systems, methods, computer-readable media, techniques, and methodologies for associating regression test cases with corresponding test fingerprints via a translation table. The regression test cases may be executed with respect to a System Under Test (SUT). While example embodiments may be described herein with respect to SUTs that are software systems such as a body of source code, it should be appreciated that the SUT may be a hardware system or a collection of hardware logic as well. Further, certain terminology used herein to describe a collection of elements, a type of data structure, or the like are not intended to be limiting in any way and are intended to encompass any variation or alternative that achieves the same function. For instance, reference made herein to a test vector or some variation thereof is not intended to limit embodiments of the invention to any particular type of data structure for representing a test vector. The combination of attribute-value pairs represented by a test vector can be stored/represented using any suitable type of data structure. As another example, the terms set, group, collection, and the like are used interchangeably herein to connote any collection of elements that may or may not be ordered. In addition, while example embodiments may be described herein in connection with a translation table for storing associations between attribute seeds of test cases and corresponding test fingerprints, it should be appreciated that the translation table may be a database, repository, or any other organized storage area.

In example embodiments, a test case may be executed and a code path traversed during execution may be determined. A collection of breakpoints encountered during traversal of the code path may then be determined and a test fingerprint indicative of code coverage of the test case may be generated based on the collection of breakpoints encountered. A translation table may then be populated with an entry that associates the test case with its corresponding test fingerprint. In example embodiments, the translation table entry may store an attribute seed of the test case in association with the test fingerprint. The attribute seed may be a representation of the combination of attribute-value pairs tested by the test case. In particular, inputs to a SUT such as a body of source code can be modeled as a collection of attributes, each of which is capable of taking on a respective one or more attribute values. CTD techniques may be employed to generate CTD test vectors, where each CTD test vector is representative of a particular combination of attribute-value pairs. A corresponding test case may be generated for each such CTD vector that tests the particular combination of attribute-value pairs embodied by the CTD vector.

Breakpoints are points in the execution of source code that occur before or after one or more lines of source code and that can halt execution of the program or trigger specific tools. In example embodiments, as a test case is executed, a code path may be traversed through a body of source code being tested, potentially resulting in multiple breakpoints being encountered, one or more of which may be encountered multiple times. As used herein, a code path may refer to a path traversed during execution of at least a portion of source code and may indicate a number of times each breakpoint is encountered, and potentially, an order in which such breakpoints are encountered during execution.

In example embodiments, a test fingerprint may be generated that is indicative of each breakpoint encountered as part of traversal of a code path during execution of a test case. The test fingerprint may be further indicative of a number of times each such breakpoint is encountered as well as, potentially, an order in which the breakpoint encounters occur. Various algorithms may be employed to generate a test fingerprint for a test case based on breakpoints encountered during execution of the test case. Such algorithms are described in co-pending application entitled “Regression Test Fingerprints Based on Breakpoint Values.”

Regardless of the particular algorithm employed to generate a test fingerprint, the fingerprint may be reversible to ascertain which breakpoints were encountered during execution of the test case to which the fingerprint corresponds and a respective number of times each such breakpoint was encountered. Further, in example embodiments, certain fingerprint generation algorithms may generate a test fingerprint that is reversible to further determine an order in which the breakpoint encounters occurred. More specifically, in example embodiments, a test fingerprint may be a data structure that is indexed with respect to the breakpoints present in a SUT. For instance, each element of a test fingerprint may correspond to a particular breakpoint. In example embodiments, a value greater than zero at a particular indexed location in the test fingerprint may indicate that the breakpoint that corresponds to that indexed location was encountered during execution of the corresponding test case. Further, in example embodiments, any given value in a test fingerprint may be broken down into constituent breakpoint values that may be indicative of a number of times a corresponding breakpoint was encountered, and potentially, an order in which the breakpoint was encountered.

In any case, in example embodiments, the test fingerprint that is generated based on the collection of breakpoints encountered during a code path traversal associated with execution of a test case may be indicative of an extent of code coverage associated with the test case. As used herein, the term code coverage refers to the portion of a SUT (e.g., a body of source code) that is tested by a test case in relation to the entire SUT (e.g., the entire body of source code). In example embodiments, a test fingerprint may reveal which breakpoints are encountered during execution of a test case, and thus, the extent of code coverage provided by the test case.

In example embodiments, a translation table that maps test cases to their corresponding breakpoint value-based test fingerprints can be used to achieve desired code coverage metrics during regression testing. For instance, if a certain amount of cumulative code coverage is desired for a regression bucket of test cases, the translation table can be accessed to identify a respective test fingerprint corresponding to each test case (based on the stored associations between the attribute seeds of test cases and the test fingerprints), and the test fingerprints can be evaluated to determine how many breakpoints were encountered during execution of each test case, and thus, a respective code coverage of each test case. A subset of test cases can then be selected that provides a cumulative code coverage that is at least as great as the desired cumulative code coverage. The cumulative code coverage associated with a set of test cases may be determined, for example, based on the total number of unique breakpoints encountered during execution of the set of test cases, as determined from the respective collection of breakpoints embodied in each test fingerprint.

In certain example embodiments, the subset of test cases to execute may be selected so as to satisfy one or more constraints. For instance, in example embodiments, the subset of test cases may be selected such that all test cases can be executed within a time constraint (e.g., a threshold period of time). As another non-limiting example, a maximum number of test cases may be specified. In such example embodiments in which one or more constraints are applicable, the subset of test cases selected may be that subset which satisfies the constraint(s) and which provides the most extensive cumulative code coverage (as determined from corresponding test fingerprints accessible via the translation table) as compared to each other candidate subset of test cases that satisfies the constraint(s).

In addition, in example embodiments, code coverage ascertained from test fingerprints accessible via the translation table may be used to determine which set of regression test cases to execute, where each set of test cases provides a desired amount of coverage of a test space. For instance, if an entire test space is modeled as the Cartesian product of all possible attribute-value pair combinations, it may be desirable to perform a reduction of the entire test space to obtain a reduced set of CTD vectors that provides a desired complete n-wise coverage of the test space but not complete coverage. As a non-limiting example, an entire test space can be reduced down to a set of CTD vectors that provides complete pairwise coverage of the test space but only partial higher-order coverage. As used herein, if a set of CTD vectors/test cases is referred to as providing complete n-wise coverage, this indicates that the set, in the aggregate, includes all possible n-way combinations of attribute values.

Often, there are multiple unique reduced sets of CTD vectors (and thus sets of corresponding test cases) that each provide complete n-wise coverage of a test space. In example embodiments, the cumulative code coverage for each of the multiple unique reduced sets of test cases—as determined from corresponding test fingerprints accessible via the translation table—can be evaluated to determine which reduced set of test cases to select for execution. For instance, in example embodiments, a respective attribute seed may be determined for each test case in a first reduced set of test cases and each test case in a second reduced set of test cases, where each of the first reduced set and the second reduced set provide complete n-wise (e.g., pairwise) coverage of a test space. The attribute seeds may be used to search a translation table for correspondingly stored test fingerprints. Each retrieved test fingerprint may be deconstructed to obtain the corresponding collection of breakpoints encountered during execution of a corresponding test case. From these collections of breakpoints, a cumulative code coverage associated with the first reduced set of test cases and a cumulative code coverage associated with the second reduced set of test cases may be determined. The reduced set of test cases associated with the more extensive cumulative code coverage may then be selected for execution in lieu of the other reduced set of test cases, or in the alternative, prioritized for execution over the other reduced set of test cases, particularly in those example scenarios involving constraint(s) on the execution of test cases.

Example embodiments of the invention also contemplate other uses for a translation table as described herein. In particular, storing associations between attribute seeds of test cases and corresponding test fingerprints provides not only a means for recording the scope of code coverage for a given test case, but also a mechanism for mapping test cases to specific areas of code and/or specific code modules covered by the test cases. Thus, in example embodiments, if a function of a SUT (e.g., a routine in a body of source code) is changed, specialized test cases targeted to the area of interest can be selected and/or regenerated as needed based on an identification of which test cases correspond to test fingerprints that identify breakpoints associated with the area of interest, thereby enabling creation of a more effective regression test suite. In example embodiments in which one hundred percent code coverage is not sought to be achieved, such specialized test cases that target specific areas of interest in a SUT may be prioritized for execution.

Example embodiments of the invention provide various technical effects over conventional regression testing methodologies. In particular, example embodiments of the invention provide a mechanism for mapping test cases to test fingerprints indicative of the extent of code coverage of the test cases. This mechanism yields the technical effect of enabling test cases to be generated or selected based on the extent of their code coverage, and thereby enables more efficient and more targeted regression testing to be performed, thus representing an improvement to computer-based regression testing technology. In example embodiments, the test fingerprints to which test cases are mapped are reversible to identify specific breakpoints encountered on traversals of code paths during execution of the test cases. The breakpoints are linked to specific areas of a SUT such as specific lines of code in a body of source code. Thus, example embodiments of the invention also provide a mechanism for mapping test cases to specific areas and modules in a SUT, which yields the technical effect of enabling selection and/or generation of specialized test cases that target specific areas of interest in a SUT (e.g., a module or function that has been changed) based on a determination that the corresponding test fingerprint is indicative of breakpoints associated with the areas of interest. This technical effect is a further improvement to computer-based regression testing technology. The above-mentioned technical effects are achieved at least in part by various technical features of example embodiments of the invention including, without limitation, determination of a code path traversed during execution of a test case; determination of a collection of breakpoints encountered during traversal of the code path; generation of a test fingerprint based on the collection of breakpoints encountered; and storage of the test fingerprint or some other identifier thereof in association with an attribute seed of the corresponding test case.

Illustrative methods in accordance with example embodiments of the invention and corresponding data structures (e.g., program modules, engines, etc.) for performing the methods will now be described. It should be noted that each operation of any of the methods300-500respectively depicted inFIGS. 3-5may be performed by one or more of the program modules or the like depicted inFIG. 1orFIG. 6, whose operation will be described in more detail hereinafter. These program modules may be implemented in any combination of hardware, software, and/or firmware. In certain example embodiments, one or more of these program modules may be implemented, at least in part, as software and/or firmware modules that include computer-executable instructions that when executed by a processing circuit cause one or more operations to be performed. A system or device described herein as being configured to implement example embodiments may include one or more processing circuits, each of which may include one or more processing units or nodes. Computer-executable instructions may include computer-executable program code that when executed by a processing unit may cause input data contained in or referenced by the computer-executable program code to be accessed and processed to yield output data.

FIG. 1is a schematic hybrid data flow/block diagram illustrating associating test cases with their corresponding breakpoint value-based test fingerprints via a translation table in accordance with one or more example embodiments of the invention.FIG. 3is a process flow diagram of an illustrative method300for associating a test case with a test fingerprint indicative of code coverage of the test case in accordance with one or more example embodiments of the invention.FIG. 3will be described in conjunction withFIG. 1hereinafter.

Referring first toFIG. 3, at block302of the method300, a test case100may be executed by a test execution engine102. In example embodiments, computer-executable instructions of a test case selection engine120may be executed to select the test case100from a set of CTD test cases108stored in one or more datastore(s)106. The selected test case100may then be provided to the test execution engine102for execution. Each CTD test case may have been generated from a corresponding CTD test vector representative of a particular combination of attribute-value pairs to be tested on a SUT such as a body of source code110. In certain example embodiments, the set of CTD test cases108may provide complete coverage of an entire test space, that is, all possible combinations of attribute values. Generally, however, it is infeasible to execute within any reasonable constraint (e.g., time constraint) the total number of CTD test cases required to provide complete coverage of an entire Cartesian product test space. Thus, in example embodiments, the set of CTD test cases108may include reduced set(s) of test cases that provide complete n-wise coverage of a test space but only partial higher-order coverage.

In example embodiments, at block304of the method300, computer-executable instructions of one or more fingerprint determination modules104of the test execution engine102may be executed to determine a code path traversed during execution of the test case100. Then, at block306of the method300, the fingerprint determination module(s)104may be executed to determine a collection of breakpoints encountered during traversal of the code path. At block308of the method300, computer-executable instructions of the fingerprint determination module(s)104may be executed to determine a test fingerprint112corresponding to the executed test case100based on the collection of breakpoints encountered during traversal of the code path.

In example embodiments, traversal of a code path through a body of source code being tested by a test case is likely to result in encountering multiple breakpoints, one or more of which may be encountered multiple times. In example embodiments, the test fingerprint112generated at block308is indicative of each breakpoint encountered and a number of times each such breakpoint is encountered as part of traversal of a code path during execution of the selected test case100. The test fingerprint112may be potentially further indicative of an order in which the breakpoint encounters occur. Any of the algorithms previously referenced may be employed to generate the test fingerprint112.

Regardless, however, of the particular algorithm employed to generate the test fingerprint112, the fingerprint112may be reversible to ascertain which breakpoints were encountered during execution of the test case100to which the fingerprint112corresponds and a respective number of times each such breakpoint was encountered. Moreover, in example embodiments, certain fingerprint generation algorithms may generate a test fingerprint that is reversible to further determine an order in which the breakpoint encounters occurred. In any case, in example embodiments, the test fingerprint112that is generated at block308based on the collection of breakpoints encountered during a code path traversal associated with execution of the selected test case100may be indicative of an extent of code coverage associated with the test case. In particular, in example embodiments, a test fingerprint may reveal which breakpoints are encountered during execution of a test case, and thus, the extent of code coverage provided by the test case.

Referring again toFIG. 3, at block310of the method300, an attribute seed114associated with the test case100may be determined. The attribute seed114may be a representation of the combination of attribute-value pairs tested by the test case100. More specifically, the attribute seed114may be a Javascript Object Notation (JSON) mapping of attributes to particular attribute values or any other suitable representation of a particular combination of attribute values. At block312of the method300, computer-executable instructions of one or more translation table modules116may be executed to store an association between the attribute seed114and the test fingerprint112. In example embodiments, a translation table118may be populated with an entry that associates the test case100with its corresponding test fingerprint112, or more specifically, that associates the attribute seed114of the test case100with the test fingerprint112.

FIG. 2is a schematic depiction of entries in an example translation table200in accordance with one or more example embodiments of the invention. The example translation table200is depicted as including multiple entries that respectively store attribute seeds202of test cases in association with corresponding test fingerprints204of the test cases. In particular, each translation table entry associates an attribute seed of a particular test case with a corresponding test fingerprint of that test case. As previously noted, each attribute seed represents a particular combination of attribute values tested by a corresponding test case. Each attribute seed is stored in the translation table200in association with a test fingerprint that is indicative of an extent of code coverage associated with execution of the test case corresponding to the attribute seed.

In example embodiments, a translation table that maps test cases to their corresponding breakpoint value-based test fingerprints such as the example translation table200can be used to achieve desired code coverage metrics during regression testing.FIG. 4depicts an example method400for utilizing the information contained in a translation table that associates attribute seeds of test cases with corresponding test fingerprints to achieve desired code coverage metrics. In particular,FIG. 4is a process flow diagram of an illustrative method400for identifying and executing a subset of test cases that satisfies constraint(s) and that has a greatest cumulative code coverage among candidate subsets of test cases that satisfy the constraint(s) in accordance with one or more example embodiments of the invention.

At block402of the method400, computer-executable instructions of the test selection engine120may be executed to determine, based on one or more constraints, that an entire regression bucket of test cases cannot be executed within the constraint(s). Example constraints include, without limitation, a time constraint specifying that regression testing must be performed within a threshold period of time; a size constraint that specifies an upper threshold on the number of regression tests that can be executed; and so forth. Thus, in example embodiments, determining that a regression bucket of test cases cannot be executed within constraint(s) may include determining that the entire set of test cases cannot be executed within a threshold period of time and/or that the regression bucket includes more than a threshold number of test cases eligible for execution.

At block404of the method400, computer-executable instructions of the test selection engine120may be executed to determine code coverage associated with the test cases in the regression bucket (e.g., at least some of the test cases108) based on corresponding test fingerprints accessed via the translation table118. In particular, attribute seeds of the test cases in the regression bucket may be used to locate corresponding test fingerprints stored in the translation table118in association with the attribute seeds. The test fingerprints can be evaluated to determine how many breakpoints were encountered during execution of each test case, and thus, a respective code coverage of each test case.

At block406of the method400, computer-executable instructions of the test case selection engine120may be executed to select a particular subset of test cases capable of being executed with the constraint(s) and having a cumulative code coverage that is greater than each other subset of test cases capable of being executed within the constraint(s). For instance, in example embodiments, the corresponding test fingerprints of each subset of test cases capable of being executed within the constraint(s) may be evaluated to determine the cumulative code coverage of each such subset. Then, at block406, the particular subset that provides the largest code coverage may be selected. Finally, at block408of the method400, the selected subset of test cases may be executed within the specified constraint(s).

In certain example embodiments, a threshold amount of cumulative code coverage may be desired. In such example embodiments, a subset of test cases may be selected that provides a cumulative code coverage that is at least as great as the desired cumulative code coverage. In this manner, the desired threshold amount of cumulative code coverage may operate as a constraint. In those example embodiments in which other constraints are present such as a time constraint, the particular subset of test cases selected for execution may provide at least the desired threshold amount of code coverage while still being capable of completing execution within the time constraint, but may not necessarily provide the largest cumulative code coverage among all subsets capable of being executed within the time constraint. This may be the case if specific areas of interest (e.g., specific portions of the source code110) are being targeted.

In addition, in example embodiments, code coverage ascertained from test fingerprints accessible via the translation table118may be used to determine which set of regression test cases to execute, where each set of test cases provides a desired amount of coverage of a test space.FIG. 5is a process flow diagram of an illustrative method500for selecting a reduced set of test cases for execution based on the cumulative code coverage of the reduced set of test cases in accordance with one or more example embodiments of the invention.

At block502of the method500, computer-executable instructions of the test case selection engine120may be executed to identify a first reduced set of test cases and a second reduced set of test cases, each of which provides desired n-wise coverage of a test space. In example embodiments, if an entire test space is modeled as the Cartesian product of all possible combinations of attribute values, it may be desirable to perform a reduction of the entire test space to obtain a reduced set of CTD vectors that provides a desired complete n-wise coverage of the test space but not complete coverage. As a non-limiting example, an entire test space can be reduced down to a set of CTD vectors that provides complete pairwise coverage of the test space but only partial higher-order coverage. Thus, in example embodiments, the first reduced set of test cases and the second reduced set of test cases may each provide complete pairwise coverage of the test space, but only partial 3-wise coverage, 4-wise coverage, and so forth. In addition, there are oftentimes multiple unique reduced sets of CTD vectors (and thus sets of corresponding test cases) that each provide complete n-wise coverage of a test space. Thus, in example embodiments, the first reduced set of test cases and the second reduced set of test cases may each constitute a unique combination of test cases. In particular, in example embodiments, the first reduced set of test cases may include at least one test case not present in the second reduced set of test cases or vice versa.

At block504of the method500, computer-executable instructions of the test case selection engine120and/or computer-executable instructions of the translation table module(s)116may be executed to access the translation table118to determine a first set of test fingerprints respectively corresponding to the first reduced set of test cases and a second set of test fingerprints respectively corresponding to the second reduced set of test cases. Then, at block506of the method500, a first cumulative code coverage associated with the first set of test fingerprints and a second cumulative code coverage associated with the second set of fingerprints may be determined. In particular, in example embodiments, a respective attribute seed may be determined for each test case in the first reduced set of test cases and each test case in the second reduced set of test cases, which may be then used to search the translation table118for correspondingly stored test fingerprints. Each retrieved test fingerprint may be deconstructed to obtain the corresponding collection of breakpoints encountered during execution of a corresponding test case. From these collections of breakpoints, the first cumulative code coverage associated with the first reduced set of test cases and the second cumulative code coverage associated with the second reduced set of test cases may be determined.

At block508of the method500, computer-executable instructions of the test case selection engine120may be executed to determine, for example, that the first cumulative code coverage is greater than the second cumulative code coverage. That is, it may be determined at block508that the first set of test fingerprints (and thus the first reduced set of test cases) provides more extensive coverage of the source code110than the second set of test fingerprints (and thus the second reduced set of test cases). Upon making this determination, computer-executable instructions of the test case selection engine120may be executed at block510of the method500to select the first reduced set of test cases for execution in lieu of the second reduced set of test cases, or in the alternative, to prioritize the first reduced set of test cases for execution over the second reduced set of test cases such as in example scenarios involving constraint(s) on the execution of test cases.

One or more illustrative embodiments of the invention are described herein. Such embodiments are merely illustrative of the scope of this invention and are not intended to be limiting in any way. Accordingly, variations, modifications, and equivalents of embodiments disclosed herein are also within the scope of this invention.

FIG. 6is a schematic diagram of an illustrative computing device600configured to implement one or more example embodiments of the invention. The illustrative computing device600may include a server or collection of servers; a personal computer; a handheld device; or the like. While the computing device600and/or any particular component of the computing device600may be described herein in the singular, it should be appreciated that multiple instances of the computing device600and/or any component thereof may be provided and functionality described in connection with the computing device600may be distributed across multiple computing devices600and/or across multiple instances of any particular component of the computing device600.

In an illustrative configuration, the computing device600may include one or more processors (processor(s))602, one or more memory devices604(generically referred to herein as memory604), one or more input/output (“I/O”) interfaces606, one or more network interfaces608, and data storage612. The computing device600may further include one or more buses610that functionally couple various components of the computing device600.

The data storage612may include removable storage and/or non-removable storage including, but not limited to, magnetic storage, optical disk storage, and/or tape storage. The data storage612may provide non-volatile storage of computer-executable instructions and other data. The memory604and the data storage612, removable and/or non-removable, are examples of computer-readable storage media (CRSM) as that term is used herein.

The data storage612may store computer-executable code, instructions, or the like that may be loadable into the memory604and executable by the processor(s)602to cause the processor(s)602to perform or initiate various operations. The data storage612may additionally store data that may be copied to memory604for use by the processor(s)602during the execution of the computer-executable instructions. Moreover, output data generated as a result of execution of the computer-executable instructions by the processor(s)602may be stored initially in memory604and may ultimately be copied to data storage612for non-volatile storage.

More specifically, the data storage612may store one or more operating systems (O/S)614; one or more database management systems (DBMS)616configured to access the memory604and/or one or more external datastores624; and one or more program modules, applications, engines, managers, computer-executable code, scripts, or the like such as, for example, a test case selection engine618, a test execution engine620, and one or more translation table modules622. The test execution engine620may include one or more sub-modules such as one or more fingerprint determination modules620A. Any of the components depicted as being stored in data storage612may include any combination of software, firmware, and/or hardware. The software and/or firmware may include computer-executable instructions (e.g., computer-executable program code) that may be loaded into the memory604for execution by one or more of the processor(s)602to perform any of the operations described earlier in connection with correspondingly named modules/engines depicted inFIG. 1.

Although not depicted inFIG. 6, the data storage612may further store various types of data utilized by components of the computing device600(e.g., data stored in the datastore(s)624). Any data stored in the data storage612may be loaded into the memory604for use by the processor(s)602in executing computer-executable instructions. In addition, any data stored in the data storage612may potentially be stored in the external datastore(s)624and may be accessed via the DBMS616and loaded in the memory604for use by the processor(s)602in executing computer-executable instructions.

Referring now to other illustrative components depicted as being stored in the data storage612, the O/S614may be loaded from the data storage612into the memory604and may provide an interface between other application software executing on the computing device600and hardware resources of the computing device600. More specifically, the O/S614may include a set of computer-executable instructions for managing hardware resources of the computing device600and for providing common services to other application programs. In certain example embodiments, the O/S614may include or otherwise control the execution of one or more of the program modules, engines, managers, or the like depicted as being stored in the data storage612. The O/S614may include any operating system now known or which may be developed in the future including, but not limited to, any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system.

The DBMS616may be loaded into the memory604and may support functionality for accessing, retrieving, storing, and/or manipulating data stored in the memory604, data stored in the data storage612, and/or data stored in external datastore(s)624. The DBMS616may use any of a variety of database models (e.g., relational model, object model, etc.) and may support any of a variety of query languages. The DBMS616may access data represented in one or more data schemas and stored in any suitable data repository. Data stored in the datastore(s)624may include, for example, breakpoint values; fingerprints; source code; regression tests; attribute seeds; translation table entries; and so forth, any portion of which may alternatively or additionally be stored in the data storage612. External datastore(s)624that may be accessible by the computing device600via the DBMS616may include, but are not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed datastores in which data is stored on more than one node of a computer network, peer-to-peer network datastores, or the like. In example embodiments, the datastore(s)624may include the datastore(s)106depicted inFIG. 1.

Referring now to other illustrative components of the computing device600, the input/output (I/O) interface(s)606may facilitate the receipt of input information by the computing device600from one or more I/O devices as well as the output of information from the computing device600to the one or more I/O devices. The I/O devices may include any of a variety of components such as a display or display screen having a touch surface or touchscreen; an audio output device for producing sound, such as a speaker; an audio capture device, such as a microphone; an image and/or video capture device, such as a camera; a haptic unit; and so forth. Any of these components may be integrated into the computing device600or may be separate. The I/O devices may further include, for example, any number of peripheral devices such as data storage devices, printing devices, and so forth.

It should be appreciated that the program modules/engines depicted inFIG. 6as being stored in the data storage612are merely illustrative and not exhaustive and that processing described as being supported by any particular module may alternatively be distributed across multiple modules, engines, or the like, or performed by a different module, engine, or the like. In addition, various program module(s), script(s), plug-in(s), Application Programming Interface(s) (API(s)), or any other suitable computer-executable code hosted locally on the computing device600and/or other computing devices may be provided to support functionality provided by the modules depicted inFIG. 6and/or additional or alternate functionality. Further, functionality may be modularized in any suitable manner such that processing described as being performed by a particular module may be performed by a collection of any number of program modules, or functionality described as being supported by any particular module may be supported, at least in part, by another module. In addition, program modules that support the functionality described herein may be executable across any number of cluster members in accordance with any suitable computing model such as, for example, a client-server model, a peer-to-peer model, and so forth. In addition, any of the functionality described as being supported by any of the modules depicted inFIG. 4may be implemented, at least partially, in hardware and/or firmware across any number of devices.

One or more operations of the method400and/or the method500may be performed by one or more computing devices600having the illustrative configuration depicted inFIG. 6, or more specifically, by one or more program modules, engines, applications, or the like executable on such a device. It should be appreciated, however, that such operations may be implemented in connection with numerous other device configurations.

Although specific embodiments of the invention have been described, one of ordinary skill in the art will recognize that numerous other modifications and alternative embodiments are within the scope of the invention. For example, any of the functionality and/or processing capabilities described with respect to a particular system, system component, device, or device component may be performed by any other system, device, or component. Further, while various illustrative implementations and architectures have been described in accordance with embodiments of the invention, one of ordinary skill in the art will appreciate that numerous other modifications to the illustrative implementations and architectures described herein are also within the scope of this invention. In addition, it should be appreciated that any operation, element, component, data, or the like described herein as being based on another operation, element, component, data, or the like may be additionally based on one or more other operations, elements, components, data, or the like. Accordingly, the phrase “based on,” or variants thereof, should be interpreted as “based at least in part on.”